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Description 

[0001 ] The present invention relates to digitalwatermarking of multimedia data and particularly video data More par-' 
ticularly. the invention concerns watermarking of compressed video data, such as JPEG and MPEG, and the detection 
of a watermark in compressed video that has been subject to geometric distortion. 

(0002] The proliferation of digitized media such as image, video and multimedia is creating a need for a security sys- 
tem which facilitates the identification of the source of the media. The expected introduction of digital video disks (DVD) 
in mass markets will further exacerbate the problem. 

[0003] Content providers, i.e. owners of works in digital data form, have a need to embed signals into 
video/image/multimedia data, including audio data, which can subsequently be detected by software and/or hardware 
devices for purposes of authenticating copyright ownership, control and management. 

[0004] There are systems for inserting a watermark into digital media and for extracting a watermark from water- 
marked media. Typical systems may comprise 

' spread spectrum watermarking by embedding a watermark signal in perceptually significant regions of an image 
tor the purposes of identifying content owner and/or possessor. An articled entitled "Secure Spread Watermarking 
tor Multimedia" by Cox et al available at http.-www.neci.nj.nec.com.tr/index.html (Technical Report No 95-110) 
describes spread spectrum watermarking which embeds a pseudo-random noise sequence into digital data for 
watermarking purposes. 

[0005] The above watermark extraction methodology requires the original image spectrum be subtracted from the 
watermarked image spectrum. This restricts the use of the method when there is no original image or orioinal imaoe 
spectrum available to the decoder. a y 

[0006] In US Patent No. 5.319,735 by R.D. Preuss et al entitled "Embedded Signaling" digital information is encoded 
to produce a sequence of code symbols. The sequence of code symbols is embedded in an audio signal by generating 
a corresponding sequence of spread spectrum code signals representing the sequence of code symbols The fre- 
qU Ih Cy « C u mP ° nentS ° f thS COde Si9 " al bein9 essentiallv confined to a preselected signaling band lying within the band- 
width of the audio signal and successive segments of the code signal corresponds to successive code symbols in the 
I 8q H em !?'J he aUd '° S ' gnal iS continuous| y frequency analyzed over a frequency band encompassing the signaling 
Dand and the code signal is dynamically filtered as a function of the analysis to provide a modified code signal with fre- 
quency component levels which are. at each time instant, essentially a preselected proportion of the levels of the audio 
signal frequency components in corresponding frequency ranges. The modified code signal and the audio signal are 
combined to provide a composite audio signal in which the digital information is embedded. This component audio sig- 
nal is then recorded on a recording medium or is otherwise subjected to a transmission channel. Two key elements of 
h.s process are the spectral shaping and spectral equalization that occur at the insertion and extraction stages respec- 
nm^i by all0Win9 embedded si 9 nal t0 be extracted without access to the unwatermarked original data. 
[0007] In a system for a spread spectrum watermark for embedded signaling, a watermark or embedded data are 
m^S** fr ° m watermarked ima 9 es or v| deo without using an original or unwatermarked version of the data. 
[0008] This method of watermarking an image or image data for embedding signals requires that the DCT (discrete 
cosine transform) and its inverse of the entire image be computed. There are fast algorithms for computing the DCT in 
'°? ™ wn ere N is the number of pixels in the image. However, for A/=512x512, the computational requirement 
is SM high, particularly if the encoding and extracting processes must occur at video rates, i.e. 30 frames per second 
This method requires approximately 30 times the computation needed for MPEG-II decompression. 
[0009] One possible way to achieve real-time video watermarking is to only watermark every N th frame. However con- 
tent owners wish to protect each and every video frame. Moreover, if it is known which frames contain embedded sig- 
nals, it is simple to remove those frames with no noticeable degradation in the video signal. 
[001 0] An alternative option is to insert the watermark into nxn blocks of the image (subimages) where n«N If the 
block aze is chosen to be 8x8. i.e. the same size as that used for MPEG image compression, then it is possible to tightly 
couple the watermark insertion and extraction procedures to those of the MPEG compression and decompression algo- 
rithms. Considerable computational saving can then be achieved since the most expensive computations relate to the 
calculation of the DCT and its inverse and these steps are already computed as part of the compression and decom- 
pression algorithm. The incremental cost of watermarking is then very small, typically less than five percent of the com- 
putational requirements associated with MPEG. 

E! 71,18 SySt6m may be advanced bv u »ng MPEG/JPEG coefficients to encode the image data 
[001 2] Furthermore, watermark information may be stored into and extracted from subimages. 
[001 3] A review of watermarking is found in an article by Cox et al.. entitled "A review of watermarking and the impor- 
tance of perceptual modeling" in Proc. of El'97, vol. 30-16. February 9-14. 1997. 

[001 4] To allow for computationally efficient detection of the watermark iii both the spatial and DCT domains, a water- 
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mark is inserted into sums of groups of 8x8 blocks in the DCT. The advantage of this approach is that, if the image is 
only available in the spatial domain, then the summation can also be performed in the spatial domain to compute a 
small number of 8x8 blocks and only these blocks must then be transformed into the DCT domain. This is because the 
sum of OCT blocks is equal to the DCT of the sum of $patial blocks. Since the computational cost of detecting water- 
s marks is now dominated by the cost of summation, the cost of detecting in the DCT and spatial domains is approxi- 
mately the same. 

[001 5] Although these watermarking techniques are highly successful, there is one problem remaining. Namely, even 
small geometric changes to an image or video frame significantly affects the DCT coefficients. For example, consider 
an image that is divided into 8 x 8 blocks and transformed into the frequency domain by computing the DCT of the 
10 blocks. If the image is now reduced in size by 1/8, then each original 8x8 pixel block is reduced to a set of 7 x 7 pixels. 
If the reduced or scaled image is again divided into 8x8 blocks and the corresponding DCTs of the blocks are calcu- 
lated, these new DCT coefficients are usually very different from those of the original unsealed image. As a result, 
watermark detection often fails. 

[0016] The present invention solves the problem of geometric scaling or affine distortion of images or video frames 
is by approximating the scale change or distortion by a spatially varying translation of each 8x8 block. The image is 
divided into a disjoint set of blocks, each block within a set experiencing the same translation. The set of blocks is dif- 
ferent for different sets. The watermark is extracted for all translations of the blocks and the maximum correlator output 
is tested for statistical significance in order to determine whether a watermark is present in the image or video frame. 
[0017] A watermark algorithm divides an image or video data into 8x8 pixel blocks. The blocks are then apportioned 
20 among 38 groups. The watermark detection process begins by summing all 8 x 8 blocks in a group, thereby reducing 
the dimension of the data into 38 blocks of 8 x 8 values each. A subset of coefficients in each of the 8 x 8 biocks is fur- 
ther summed together to form a one-dimensional vector of length 38. 

[0018] The 38 groups of 8 x 8 blocks of DCT coefficients are chosen in such a way that translation shifts of multiples 
of 8 pixels simply result in a cyclic rotation of the extracted 1-D watermark vector. For shifts that are not a multiple of 8, 

25 it is only necessary to determine the translation remainder modulo 8, i.e. translation of 0 to 7 pixels in both the x and y 
directions. This is accomplished by taking the 38-8 x 8 blocks of DCT coefficients and first transforming them to the spa- 
tial domain. Next, the data is shifted by one of the 64 possible translations before being transformed back to the DCT 
domain. The watermark is then extracted and tested as will be described below. The process is repeated for all trans- 
lations and the maximum value of the correlator output is then tested for statistical significance in order to determine 

30 whether a watermark is present 

[001 9] Since the amount of translation of the image or video frame is not known beforehand, by repeating the shift for 
all translations of 0 to 7 pixels, and then determining the signal with the highest correlation to a watermark, the most 
likely translation factor and watermark are determined. 

[0020] The present invention approximates geometric distortions by spatially varying translations. While removing the 
35 change or distortion from the image or video data is extremely difficult manipulation in the form of spatial translation of 
blocks is a relatively simple way to extract a watermark from the distorted image or video, without restoring the image 
or video to its original, undistorted condition. 

[0021] A principal object of thfe present invention is therefore, the provision of a watermark detection method which 
corrects for affine distortion of the watermarked data, such as image or video data. 
40 [0022] Further objects of the invention will become more clearly apparent when the following description is read in 
conjunction with the accompanying drawing. 

Figure 1 is a flow diagram of a watermark detection procedure for the case where a stream of compressed video 
is input; 

45 Figure 2 is a flow diagram of a watermark detection procedure for the case where uncompressed data is the input; 
Figure 3 is a representation of a scaled sequence of blocks relative to an original sequence of blocks; 
Figure 4 is a schematic representation of a set of eight accumulator arrays; 

Figure 5 is a flow diagram of a preferred embodiment of the invention when compressed video is input; and 
Figure 6 is a flow diagram of a preferred embodiment of the invention when uncompressed video is input. 

so 

[0023] As used in the following description the terms image, image data, video, video frame and video data will be 
understood to apply equally to video, image and multimedia data. The term "watermark" will be understood to include 
embedded data, symbols, images, instructions or any other identifying information. 

[0024] A significant problem in designing a watermarking system and particularly for DVD applications is to insert 
55 watermarks that are easy to detect in both MPEG compressed and uncompressed video. Most watermarks can be eas- 
ily detected in either the spatial (uncompressed) or the DCT (compressed) domains, but not both. This means that 
detecting the watermarks in one or the other domain requires conversion of the entire image, which is generally too 
computationally expensive for DVD applications. In the following description, it is assumed that a watermark was 
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inserted into the video data in a conventional manner, for example in a manner described in patent applications and 
publications cited above. 

[0025] A solution to the problem is to insert the watermark in such a way that it can be extracted from a small number 
(or groups) of 8 x 8 blocks, preferably 38 groups: Each group is the sum of many blocks in the image. If 8 x 8 blocks in 
the spatial domain are added together in a group, andlhen the DCT of the sum is computed, it is the equivalent of com- 
puting the DCT of each block and adding the DCTs together. The DCT of the sum of the 8 x 8 blocks in a group in the 
spatial domain is the equivalent of the sum of the DCT of each 8x8 block in the group. The result is that the 38 summed 
blocks can be computed directly from either uncompressed or MPEG encoded data, with conversion into the spatial or 
compressed domain being performed only after the summation is performed. Thus, rather than computing 5400 DCT 
coefficients, only 38 DCTs are computed. 

[0026] The size of blocks is preferably selected to be 8 x 8 because of the structure of common MPEG video signals. 
However, the invention is applicable to any n x n block. The number of groups arbitrarily is selected to be 38. The inven- 
tion can use other quantities of groups of blocks. Therefore, in the following description when reference is made to 38 
items, it will be understood that any predetermined number of items could be used. 

[0027] Referring now to the figures and to Figure 1 in particular, there is shown a flow diagram of a watermark detec- 
tion procedure for the case where an MPEG stream of compressed video is input. As each 8x8 block of DCT coeffi- 
cients is decoded from the input stream, it is added in one of 38 - 8 x 8 accumulators 10(1), 10(2) ...10(38). After the 
entire image has been added into the 38 • 8 x 8 accumulators, the watermark is extracted from the 38 accumulators 
resulting groups in a conventional manner 12. The extraction process is simple, requiring only about 500 addition oper- 
ations. Finally the extracted watermark is decoded 14 and statistically tested to determine if the extracted signal is a 
real watermark, or merely noise present in an unwatermarked image. If it is determined that a watermark actually is 
present, the watermark may be decoded to yield information bits necessary for DVD applications. The input video is 
added to a particular accumulator based on a predetermined mapping relationship used when the watermark was 
inserted into the video stream. 

[0028] Figure 2 shows the corresponding watermarking detection procedure for the case where the input is a stream 
of uncompressed data or uncompressed video. As each pixel arrives, it is added to one of 38 - 8 x 8 accumulators 20 
according to a predetermined mapping algorithm corresponding to the algorithm used when inserting the watermark 
into the video. After the entire image has been added into the accumulators, the DCT values of the accumulated video 
is calculated 22 and the remainder of the watermark decoding process proceeds as described in conjunction with the 
receipt of MPEG video shown in Figure 1 . 

[0029] The general methods described above in conjunction with Figures 1 and 2 work very well as long as the par- 
titioning of the images into 8x8 blocks during distortion is identical to the partitioning used when the watermark was 
inserted. However, if the image was subjected to geometric or other affine distortion, the watermark becomes difficult 
to detect. 

[0030] In order to counteract such distortions, the above watermark detection processes are modified. Consider, an 
image that has been reduced by one-eighth in the horizontal direction. Referring to Figure 3. the lower row of boxes 
shows the spatial position of a sequence of 8 x 8 blocks, 1,2.3.... 9. in the original image and the upper row of boxes 
shows the positions of the 8 x 8 blocks. V, 2*. 3*. ... 10*. after the original image has been reduced or scaled by a scale 
factor. Block 1 in the original image has a strong overlap with block V in the scaled image. Block 2 in the original image 
shares 6 columns of 8 x 8 pixels with block 2'. having been shifted by 1 pixel relative to block 2. Thus, if the columns of 
block 2* were shifted one column to the right before computation of its DCT. the result would be quite similar to the DCT 
of the original block 2. The difference between the scaled and original blocks is entirely due to the fact that column 1 of 
block 2' coincides with column 8 in block 1 . This is a source of additional noise for the watermark detector. 
[0031 ] Block 1 ' in the scaled image only differs by a single column of pixels from its corresponding block 1 in the orig- 
inal image. However. Block 5* only has 4 columns in common with block 4 in the original image. Moreover, 4 columns 
of block 5' are not common to block 4 and 4 columns of block 4 are not common to block 5'. Consequently, block 5' is a 
very noisy approximation to block 4, even after compensation for the relative shift in translation. 
[0032] Considered in this manner, scale changes can be approximated by a spatially varying translation of the 8 x 8 
blocks of the scaled image. Therefore the watermark extraction process is modified to achieve this result. The single 
array of 38 accumulators is replaced by a set of 8 such accumulator arrays for the one-dimension (1 -D) horizontal shifts. 
For two-dimension (2-D) translation, 8 x 8 or 64 accumulator arrays are necessary. Only 8 arrays are necessary since 
multiples of 8 result in cyclic shifts of the extracted watermark. 

[0033] Figure 4 shows a set of 8 accumulator arrays, each associated with a translation of a predetermined number 
of columns. During the accumulation phase of the extraction procedure, the 8 x 8 Block r is added to accumulator 1 of 
set 40, the 8 x 8 Block 2 is added to accumulator 2 of set 42. Block 3* to accumulator 3 of set 43. and block 4' to accu- 
mulator 4 of set 44. Block 5* shares equal amounts of data with both Blocks 4 and 5. Thus Block 5' is added to either 
accumulator 4 or 5 of set 44 depending upon the rules used to direct the blocks to respective accumulators. Block 6' is 
added to accumulator 5 of set 45 and so forth. The rules regarding where the blocks are added are a function of the 
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assumed scaling. That is, the accumulators are chosen to provide the necessary spatial translation in order to compen- 
sate for the assumed or known scale distortion. 

[0034] After completing the accumulation of the 8 x 8 blocks in each accumulator of sets 41 through 47, the summed 
data are transformed from the DCT domain into the spatial domain and shifted by one column (set 41), two columns 
(set 42) and so forth, respectively before the spatial domain data is transformed back into the DCT domain. Now. the 
relative translation^ shift has been removed and the data in the corresponding accumulators in the set of eight accu- 
mulator arrays 40 to 47 are added together to form a single (one-dimension) accumulator array of 38-8 x 8 blocks. 
Watermark extraction then proceeds as described above. 

[0035] Figure 5 shows the process when compressed video is input and Figure 6 shows the process when uncom- 
pressed video is input. 

[0036] In Figure 5, the input compressed video. e.g. MPEG video, is placed into the array 50 of accumulators 51 . Each 
accumulator 51 comprises 38 - 8 x 8 blocks. There are 64 such accumulators to compensate for two-dimensional trans- 
lation. The contents of the 64 accumulators 51 undergo an inverse transform 52 into the spatial domain and are placed 
into an array 53 of 64 accumulators 54. Each accumulator 54 comprises 38 - 8 x 8 blocks. The data in the array 53 
undergoes shifting (translation) as described above to compensate for a known or assumed geometric distortion. The 
shifted data are summed in array 55 comprising 64 accumulators 56. Each accumulator 56, in turn, comprises 38 - 8 x 
8 blocks of data in the spatial domain. The corresponding blocks in each accumulator 56 are summed together into a 
one-dimension array of 38 - 8 x 8 blocks 58. The summed data is subject to a discrete cosine transform and the DCT 
values are stored in accumulator array 60. The resulting summed DCT data in the single accumulator 60 of 38-8 x 8 
blocks is subjected to watermark extraction 1 2 and watermark decoding 1 4 as described above, in conjunction with Fig- 
ure 1. 

[0037] Experimental tests show that almost equivalent results can be obtained using only data from a subset of the 
accumulators in array 50. 53. 55. for example, using the data in the shift 0, 1, 6 and 7 accumulator arrays (40. 41, 46 
and 47) and disregarding the data in the other accumulators. This results in a significant memory saving. A further 
improvement is gained by using a coarser quantization of shifts such that the first accumulator array contains data that 
has been shifted by 0 or 1 . Similarly, accumulator array 2 handles shifts of 2. and 3. etc. 

[0038] In Figure 6, the uncompressed video is added in an array 64 of 38 - 8 x 8 accumulators 65 based on a look- 
up table 62 that sends the incoming video pixels to a particular accumulator according to a mapping corresponding to 
the assumed geometric distortion to which the watermark video has been subjected. The summed data in the accumu- 
lators 65 are converted into DCT coefficients. The DCT coefficients are summed in a one-dimension array 66 of 38 - 8 
x 8 block accumulators 67. The data in the accumulator 66 is subjected to watermark extraction and watermark decod- 
ing as described above. 

[0039] While the above description refers to horizontal or column shifts, it will be apparent to those skilled in the art 
that the same process is equally applicable to vertical or row shifts or to a combination of row and column shifts. 
[0040] The above example assumed that the reduced scale of the image is known. A different scale requires a differ- 
ent association of scaled blocks to the accumulator arrays. However, this is straightforward to compute. 
[0041] A further reduction in memory can be achieved by spreading the computation over many frames. In this man- 
ner, only a single accumulator array is needed. In frame 0 blocks with shift 0 are accumulated. At the end of frame 0 the 
accumulator shifts by 1 so that in frame 1, blocks shifted by 1 are accumulated. Thus, after 64 frames, the same result 
is obtained as earlier, requiring only 1/64 of the memory. 
[0042] The pseudocode for performing the shifts is as follows: 
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g = "scaling group" (i.e. the number of sets of 38 accumulators used) 
sx = 0 
sy = 0 

For each of g frames 

Add ONLY the blocks that must be shifted by sx pixels to the right and sy 
pixels upward into the single set of 8 x 8 accumulators. Ignore all the other 
blocks. 

Compute BDCTs on the accumulators. 
Shift the accumulators left one pixel. 

sx = (sx + I) -=-8 
if ( sx == 0) 

{ 

Shift the accumulators down one pixel. 
sy = (sy + l)-r 8 

\ 

Compute DCTs on the accumulators. 



} 



30 [0043] In practice, the scale or geometric distortion is not known, or the scale is not a integer number of columns (or 
rows), although the range of scales of interest often is known. For example, in video it is assumed that the scale may 
vary by ± 5% and is fixed for the duration of a movie (sequence of video frames), or at least for extended periods of a 
movie. This latter assumption is reasonable, since if the scale changes frequently, the perceived video quality would be 
degraded. In these circumstances, a search process must be initiated in which the watermark detector attempts to 

35 determine the amount of scaling. For the example of ± 5% in both the horizontal and vertical directions, there are 1 1 x 
1 1 possible scales, if the scaling is quantized in unit intervals. "Unit intervals" refers to the fact that if the reduction is not 
1/8. 1/4, 3/8, 1/2, 5/8, 3/4, or 7/8, then the overlap will not coincide with an integer number of rows or columns. In that 
case, the translational shift will be "rounded" to an integer number of rows or columns. If 2 seconds is needed to detect 
a watermark at a fixed scale, then 2 x 121 =* 242 seconds are needed to search all possible scales. This can be consid- 

40 ered the worst case latency. In practice, the average latency will be much less, particularly of an intelligent search pro- 
cedure is used which search most likely scales first. 

[0044] Scale is not the only geometric distortion that may be approximated by a spatially varying translation of the 8 
x 8 DCT blocks. Other affine transforms can also be handled similarly. 

45 Claims 

1 . A method for extracting a watermark from watermarked compressed data comprising the steps of: 

receiving watermarked compressed data in the form of n x n blocks of data; 
so spatially translating the blocks to compensate for geometric distortion; and 

extracting a watermark from the translated data. 

2. A method of extracting a watermark from watermarked compressed data comprising the steps of: 

55 receiving watermarked compressed data in the form of 8 x 8 blocks; 

accumulating said blocks in a set of a predetermined number of groups of 8 x 8 blocks; 
converting the accumulated blocks into spatial domain; 

spatially translating the spatial domain accumulated blocks to compensate for an assumed affine distortion; 
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summing the translated blocks; and 

extracting a watermark from the translated blocks. 

* 

A method of extracting a watermark from watermark as set forth in claim 2 where said extracting a watermark com- 
prises calculating discrete cosine transform values of the summed blocks. 

A method of extracting a watermark from watermark as set forth in claim 2 or 3 where said steps are repeated for 
different assumed aff ine distortions. 
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